Add the containerV2 flag to compute initial heap percentage using processor count, and disable setting the ActiveProcessorCount JVM option
#361
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
November 13, 2023 17:42
…ap percentage to be max(50%, (75%*heap - 3*numProcessors)/heap)
Generate changelog in
|
mpritham
changed the title
UseProcessorAwareInitialHeapPercentage flag to compute initial heap percentage using processor count, and disable setting the ActiveProcessorCount JVM optionUseProcessorAwareInitialHeapSize flag to compute initial heap percentage using processor count, and disable setting the ActiveProcessorCount JVM option
mpritham
commented
Nov 13, 2023
integration_test/testdata/launcher-custom-experimental-processor-aware-heap-pct-flag.yml
Outdated
Show resolved
Hide resolved
mpritham
commented
Nov 13, 2023
mpritham
commented
Nov 13, 2023
mpritham
commented
Nov 13, 2023
|
Converting back to draft. I'd like to add a test. |
carterkozak
reviewed
Nov 14, 2023
added 2 commits
November 14, 2023 10:47
…ct' of github.com:palantir/go-java-launcher into pmarupaka/updateActiveProcessorCountFlagAndInitialHeapPct
mpritham
commented
Nov 14, 2023
mpritham
commented
Nov 14, 2023
mpritham
commented
Nov 14, 2023
[skip ci]
carterkozak
reviewed
Nov 16, 2023
…itial/max ram percentage when memory.limit_in_bytes is not present
mpritham
changed the title
experimentalContainerV2 flag to compute initial heap percentage using processor count, and disable setting the ActiveProcessorCount JVM optioncontainerV2 flag to compute initial heap percentage using processor count, and disable setting the ActiveProcessorCount JVM option
carterkozak
reviewed
Nov 16, 2023
carterkozak
reviewed
Nov 16, 2023
carterkozak
reviewed
Nov 16, 2023
added 2 commits
November 16, 2023 10:36
… invocation after a flag check
carterkozak
reviewed
Nov 16, 2023
README.md
Outdated
| ``-XX:MaxRAMPercentage=`` nor ``-XX:InitialRAMPercentage=`` prefixes are present in either static or custom jvm opts, | ||
| ``-Xmx|-Xms`` will both be set to be 75% of the heap size minus 3mb per processor, with a minimum value of 50% of the | ||
| heap. | ||
| 1. If neither ``-XX:MaxRAMPercentage=`` nor ``-XX:InitialRAMPercentage=`` prefixes are present in either static or |
There was a problem hiding this comment.
I think we need to clarify that this case only applies when cgroup memory limits aren't detected
There was a problem hiding this comment.
It may also be fair to exclude the fallback info from the documentation as it shouldn't be reachable in rubix
There was a problem hiding this comment.
I updated it to specify that the fallback is used when we're not able to get the cgroup memory limit.
carterkozak
reviewed
Nov 16, 2023
|
👍 |
bulldozer-bot
bot
deleted the
pmarupaka/updateActiveProcessorCountFlagAndInitialHeapPct
branch
|
Released 1.93.0 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this PR
This is part of a larger effort in improving stability of Java services in Rubix.
-XX:ActiveProcessorCountk8s resource requests provide a minimum bound on resource usage. The
-XX:ActiveProcessorCountJVM option acts as an upper bound on resource usage in several internal JDK features (e.g. GC threads, compiler threads). We currently set the active processor count based on resource requests. This has led to unpredictable performance under load.Setting initial heap size to be aware of the number of processors
From the above change, we're no longer limiting the number of processors available to the JVM. In the past, when we've removed limits on the number of processors (due to JDK-8281181 being released in a critical patch update), we've encountered several OOM exceptions. This PR allocates 3mb per processor, by removing that much from the heap size. The rough 3mb figure was obtained by observation (cc: @carterkozak).
After this PR
==COMMIT_MSG==
When the experimental
containerV2is set, 1) compute the heap percentage as 75% of the heap minus 3mb per processor, with a minimum value of 50%, and 2) don't set the-XX:ActiveProcessorCountJVM option.==COMMIT_MSG==
Possible downsides?
This change will potentially waste available memory for services which would benefit from larger heap sizes. Also, the heap size heuristic is simple; a more sophisticated solution would allow for better heap utilization (see #323).